Skip to content

feat(streaming): improve locality backfill progress #23454

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Oct 21, 2025
Merged

feat(streaming): improve locality backfill progress #23454

merged 2 commits into from
Oct 21, 2025
+137 −55

Conversation

@chenzl25
Copy link
Contributor

@chenzl25 chenzl25 commented Oct 14, 2025 •
edited
Loading

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

  • Tracking: locality backfill #23338
  • Improve the ddl progress for locality backfill, since we will buffer data for locality backfill, we need to calculate and report these data to the meta node to make the progress calculation more precise.

Checklist

  • I have written necessary rustdoc comments.
  • I have added necessary unit tests and integration tests.
  • I have added test labels as necessary.
  • I have added fuzzing tests or opened an issue to track them.
  • My PR contains breaking changes.
  • My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
  • I have checked the Release Timeline and Currently Supported Versions to determine which release branches I need to cherry-pick this PR into.

Documentation

  • My PR needs documentation updates.
Release note

@chenzl25 Graphite App
Copy link
Contributor Author

chenzl25 commented Oct 14, 2025 •
edited
Loading

This stack of pull requests is managed by Graphite. Learn more about stacking.

@chenzl25 chenzl25 mentioned this pull request Oct 14, 2025
Merged
8 tasks
@chenzl25 chenzl25 changed the title improve locality backfill progress feat(streaming): improve locality backfill progress Oct 14, 2025
@github-actions github-actions bot added type/feature Type: New feature. and removed Invalid PR Title labels Oct 14, 2025
@chenzl25 chenzl25 marked this pull request as ready for review October 15, 2025 08:20
@chenzl25 chenzl25 requested review from kwannoel and wenym1 October 15, 2025 08:22
@chenzl25 chenzl25 mentioned this pull request Oct 15, 2025
Open
7 tasks
@chenzl25 chenzl25 force-pushed the dylan/improve_locality_backfill_progress branch from 01f9b68 to 9630335 Compare October 15, 2025 08:55
@chenzl25 chenzl25 force-pushed the dylan/support_truncate_locality_backfill_state branch from 7dfb8d6 to 7ff357f Compare October 15, 2025 08:55
@chenzl25 chenzl25 force-pushed the dylan/improve_locality_backfill_progress branch from 9630335 to b13d4b8 Compare October 15, 2025 10:48
@chenzl25 chenzl25 force-pushed the dylan/support_truncate_locality_backfill_state branch 2 times, most recently from 562162a to b8b1815 Compare October 15, 2025 10:50
@chenzl25 chenzl25 force-pushed the dylan/improve_locality_backfill_progress branch from b13d4b8 to a3ec73d Compare October 15, 2025 10:50
Base automatically changed from dylan/support_truncate_locality_backfill_state to main October 15, 2025 11:51
@chenzl25 chenzl25 force-pushed the dylan/improve_locality_backfill_progress branch from a3ec73d to 3b5cdbf Compare October 15, 2025 13:22
kwannoel
kwannoel reviewed Oct 21, 2025
View reviewed changes
@chenzl25 chenzl25 requested a review from kwannoel October 21, 2025 07:43
kwannoel
kwannoel approved these changes Oct 21, 2025
View reviewed changes
Copy link
Contributor

@kwannoel kwannoel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Inside the locality_provider executor I guess we don't decrement the buffered count, so it's monotonically increasing. We instead increment the consume count on the snapshot side, so that will indicate the consumption.

@chenzl25 chenzl25 enabled auto-merge October 21, 2025 08:35
@chenzl25 chenzl25 added this pull request to the merge queue Oct 21, 2025
Merged via the queue into main with commit 25dd318 Oct 21, 2025
39 of 40 checks passed
@chenzl25 chenzl25 deleted the dylan/improve_locality_backfill_progress branch October 21, 2025 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kwannoel kwannoel kwannoel approved these changes

@wenym1 wenym1 Awaiting requested review from wenym1

Assignees

No one assigned

Labels

ci/run-e2e-single-node-tests ci/run-e2e-test-other-backends type/feature Type: New feature.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@chenzl25 @kwannoel